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Abstract 


The  goal  of  the  United  States  Army  Strategic  Software  Improvement  Program  is  to  dramatically 
improve  the  acquisition  of  software-intensive  systems.  One  of  the  initiatives  undertaken  by  the 
program  is  to  begin  building  a  level  of  technical  expertise  in  modem  software  architecture  prac¬ 
tices  within  the  Army  acquisition  community. 

This  report  describes  the  Software  Architecture  Initiative  of  the  Army  Strategic  Software  Im¬ 
provement  Program.  Results  to  date  are  encouraging  and  serve  as  a  guide  for  other  acquisition 
organizations  seeking  to  strengthen  their  technical  competencies. 
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1  Introduction 


The  modem  military  increasingly  relies  on  sophisticated  and  ever  more  complex  weapons,  infor¬ 
mation,  and  communications  systems  to  perform  its  tasks.  Adding  to  that  complexity  is  the  desire 
to  create  a  network-centric  force  capable  of  employing  those  systems  in  new  and  emergent  ways 
to  achieve  unparalleled  and  unprecedented  battlefield  dominance.  The  backbone  of  these  systems 
and  capabilities  is  software.  Software  drives  the  functionality  and  performance  of  these  systems  as 
well  as  the  intricate  networks  that  tie  them  together. 

®  ... 

Since  late  2002,  the  Carnegie  Mellon  Software  Engineering  Institute  (SEI)  has  been  working 
with  the  Assistant  Secretary  of  the  Army  for  Acquisition,  Logistics,  and  Technology  (ASA(ALT)) 
in  a  strategic  partnership.  The  objective:  Improve  the  Army’s  techniques  for  acquiring  systems 
with  high  software  content,  called  software-intensive  systems,1  or  SIS.  Known  as  the  Army  Stra¬ 
tegic  Software  Improvement  Program,  or  AS  SIP,  the  program  is  devising  strategies  and  initiatives 
that  will  ultimately  enhance  the  U.S.  Army’s  ability  to  be  a  “smart  buyer”  of  software-intensive 
systems. 

One  AS  SIP  initiative  focuses  on  software  architecture.  Sound  software  architecture  practices  are  a 
strong  success  factor  in  SIS  programs.  However,  initial  investigations  into  Army  SIS  acquisition 
indicated  that  while  software  architecture  practices  were  deemed  important,  methods  and  skills  to 
carry  out  those  practices  were  perceived  to  be  inadequate.  In  response,  the  ASSIP  formulated  an 
initiative  to  build  an  organic  software  architecture  capability  within  the  Army  acquisition  com¬ 
munity. 

This  technical  report  describes  the  work  done  to  lay  the  foundation  for  an  organic  Army  software 
architecture  capability.  That  includes  training  Army  professionals  in  software  architecture  prac¬ 
tices  and  conducting  software  architecture  evaluations,  both  as  part  of  the  ASSIP  Software  Archi¬ 
tecture  Initiative  (SAI)  and  separately.  This  report  provides  an  accounting  of  the  results  and  les¬ 
sons  learned  from  the  initiative  and  related  work,  and  enables  the  launch  of  similar  approaches  in 
the  broader  acquisition  community.  The  results  of  architecture  evaluations  undertaken  by  selected 
Army  acquisition  programs  will  be  the  subject  of  other  reports. 


Carnegie  Mellon  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 

According  to  the  Defense  Acquisition  University  (DAU),  a  software-intensive  system  is  one  in  which  software 
represents  the  largest  segment  in  one  or  more  of  the  following  criteria:  system  development  cost,  system  de¬ 
velopment  risk,  system  functionality,  or  development  time  [DAU  2005], 
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2  TheASSIP 


Before  examining  the  details  of  the  SAI,  a  brief  background  on  the  ASSIP  will  provide  the  reader 
with  context.  The  ASSIP  is  a  multiyear  effort  targeted  at  dramatically  improving  the  way  in  which 
the  Army  acquires  software-intensive  systems.  To  achieve  this  goal,  the  ASSIP  works  directly 
with  the  Army’s  Program  Executive  Officers  (PEOs)  and  the  program  management  offices 
(PMOs)  that  they  oversee.  Although  sponsored  by  the  ASA(ALT),  the  ASSIP  strives  to  include 
representation  from  Army  organizations  that  are  not  necessarily  acquisition  organizations,  per  se, 
but  influence  Army  acquisitions.  This  broader  sense  of  an  Army  acquisition  community  includes 

•  the  Army  Software  Engineering  Centers  (SECs)  that  provide  software  support  for  fielded 
systems  (and,  in  some  cases,  new  development) 

•  the  Army  Test  and  Evaluation  Command  (ATEC),  which  is  responsible  for  testing  Army 
materiel  solutions 

•  the  Army  Training  and  Doctrine  Command  (TRADOC),  which  defines  the  capabilities  to  be 
addressed  by  materiel  solutions  and  also  provides  an  interface  to  the  end  users 

The  ASSIP  also  maintains  contact  with  other  organizations  such  as  the  Defense  Acquisition  Uni¬ 
versity  (DAU)  and  the  acquisition  functions  of  the  other  services  to  stay  abreast  of  developments 
and  issues  that  affect  military  acquisition  in  general. 

Organizationally,  there  are  two  main  bodies  involved  in  the  ASSIP:  the  Senior  Steering  Group 
(SSG)  and  the  ASSIP  Action  Group  (AAG).  The  SSG,  composed  of  the  Army’s  PEOs,  the  mili¬ 
tary  deputy  (MILDEP)  to  the  ASA(ALT),  and  the  director  of  the  SEI,  provides  overall  guidance 
to  the  effort.  The  AAG,  which  consists  of  representatives  from  each  of  the  PEOs  and  from  the 
SECs,  ATEC,  TRADOC,  as  well  as  SEI  technical  staff  members,  develops  and  implements  im¬ 
provement  strategies.  The  ASSIP  is  a  collaborative  partnership;  the  SSG  and  AAG  are  co-chaired 
by  the  Army  and  the  SEI.  The  SEI’s  role  includes  offering  expert  guidance  on  software  acquisi¬ 
tion  and  process  issues,  providing  secretariat  services  to  the  SSG  and  AAG,  and  acting  as  a  cata¬ 
lyst  for  change  by  serving  as  a  transition  agent  to  assist  DoD  organizations  in  applying  modem 
software  engineering  practices.  As  depicted  in  Figure  1,  the  ASSIP  is  predicated  on  the  idea  that 
better  acquisition  practices  will  lead  to  better  systems  and  overall  results. 
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Figure  1:  Continuous  Improvement  Yields  a  More  Capable  Acquisition  Function 

Early  steps  taken  by  the  ASSIP  included  efforts  to  baseline  the  current  state  of  Army  acquisition 
in  order  to  reason  about  what  sorts  of  changes  might  be  needed.  Determining  the  baseline  in¬ 
volved  a  three-phased  approach: 

1 .  Conducting  a  survey  of  Anny  acquisition  professionals  to  get  a  lay  of  the  land.  As  a  first  step 
toward  understanding  the  current  state  of  Army  acquisition,  the  survey  sought  to  provide 
preliminary  insight  into  the  major  acquisition-related  problem  areas.  Although  not  conclu¬ 
sive,  the  responses  were  invaluable  in  setting  expectations  about  the  range  of  potential  prob¬ 
lems. 

2.  Conducting  a  series  of  structured  interviews  (referred  to  as  benchmarking  for  improvement, 
or  BFI)  with  personnel  from  selected  Army  program  offices  to  discover  the  effectiveness  of 
the  business  practices  and  processes  used  within  PMOs.  Although  interviews  were  structured 
around  version  1.0  of  the  Capability  Maturity  Model  Integration  Acquisition  Module 

(C M M I  -AM)  [Bernard  2004],  the  BFI  engagements  elicited  the  processes  employed  by 
programs,  using  a  model  as  a  guideline,  instead  of  rating  program  office  processes  against  a 
model.  BFI  results  were  aggregated  in  a  non-attributable  manner  to  form  a  picture  of  Army 
acquisition  from  the  program  office  perspective. 

3.  Conducting  a  series  of  in-person  interviews  with  the  Army’s  PEOs  to  get  their  unique  per¬ 
spectives  on  the  state  of  Army  acquisition.  Each  of  the  PEOs  represented  a  wealth  of  acquisi¬ 
tion  experience.  The  interview  questions,  which  were  the  same  for  each  PEO,  were  formu¬ 
lated  to  glean  the  insights  from  that  experience.  Specifically,  the  interviews  sought  out  the 
PEOs’  overall  opinions  about  Army  acquisition,  the  activities  in  each  PEO’s  office,  and  the 
ways  in  which  the  ASSIP  could  help  improve  acquisition  of  software-intensive  systems. 

As  with  the  BFIs,  interview  results  were  aggregated  in  a  non-attributable  manner. 

Results  of  the  inquiries  are  documented  in  [Kasunic  2004],  [Keeler  2005],  and  [Blanchette  2005], 
respectively. 


CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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These  data-gathering  approaches  yielded  information  for  structuring  the  ASSIP  plan  of  attack. 
Each  fiscal  year  (FY),  the  ASSIP  produces  a  Strategic  Software  Improvement  Master  Plan 
(SSIMP)  that  delineates  the  tasks  to  be  performed  in  that  year.  The  tasks  are  aligned  to  initiatives 
that  are  based  on  the  results  of  the  initial  baselining  of  Army  acquisition  as  well  as  subsequent 
findings.  One  of  those,  the  Software  Architecture  Initiative,  is  the  subject  of  this  report. 
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3  Motivation  for  Army  Software  Architecture  Practices 


The  importance  of  software  architecture  practices  to  successful  software-intensive  programs  is 
becoming  increasingly  well  known.  The  Data  and  Analysis  Center  for  Software  (DACS)* 2  lists  an 
“architecture  first  approach”  among  its  software  acquisition  Gold  Practices  [DACS  2004],  This 
appreciation  of  the  value  of  software  architecture  practices  is  not  a  recent  epiphany;  a  Defense 
Science  Board  (DSB)  report  from  as  far  back  as  1994  called  out  the  potential  for  software  archi¬ 
tecture  and  product  line  techniques  to  reduce  cost  and  cycle  times  [DSB  1994].  Additionally,  a 
November  2000  report  from  the  DSB  highlighted  software  architecture  as  “a  central  theme  for 
software  reuse,  product  lines,  and  greater  exploitation  of  commercial  technology  and  practices” 
[DSB  2000].  In  2001,  a  U.S.  Army  lessons-leamed  workshop  focusing  on  software  upgrade  pro¬ 
grams  concluded,  in  part,  that  architecture  is  “a  key  technical  focus  for  the  system,”  noting  par¬ 
ticularly  the  criticality  of  the  architecture  in  detennining  the  future  ability  to  upgrade  the  system 
[Anderson  2001], 

Considering  the  importance  of  software  architecture  practices  to  successful  SIS  acquisition,  one 
might  have  expected  that  such  practices  were  prevalent  in  Army  (and  other  services’)  acquisition 
programs.  Such  appears  not  to  be  the  case.  In  2002,  the  Department  of  Defense  (DoD)  Tri-Service 
Assessment  Initiative  (TAI)  reported  that  poor  software  architecture  practices  was  one  of  the  sys¬ 
temic  causal  factors  of  software-intensive  systems  issues,  based  on  assessments  of  21  DoD  pro¬ 
grams  [McGarry  2002].  This  finding  suggests  that  simply  engaging  in  a  task  called  “software  ar¬ 
chitecture”  is  insufficient  to  leverage  the  benefits  of  software  architecture.  It  suggests  that  both 
acquirer  and  supplier  should  also  engage  in  an  evaluation  of  the  architecture’s  quality  and  robust¬ 
ness  to  ensure  that  it  is  living  up  to  its  potential.  Indeed,  some  larger  defense  contractors  routinely 
use  some  fonn  of  architecture  evaluation  on  many  of  their  programs  [Bass  2006]. 

Recently,  the  SEI  conducted  an  analysis  of  18  software  architecture  evaluations  perfonned  by  the 
institute  between  2000  and  2005,  including  12  for  DoD  programs.  The  evaluations  were  con¬ 
ducted  using  the  SEI  Architecture  Tradeoff  Analysis  Method®  (AT AM®). 3  The  analysis  yielded 
some  interesting  results.  For  instance,  more  than  half  of  the  evaluations  uncovered  significant 
program  risks  related  to  an  organization’s  failure  to  grasp  the  magnitude  of  the  software  architec¬ 
ture  effort,  as  manifested  by  lack  of  training,  lack  of  tools,  poor  planning,  and  other  problems 
[Bass  2006].  This  observation  supports  the  findings  of  earlier  reports:  organizations  pay  insuffi¬ 
cient  attention  to  software  architecture  practices.  Further,  nearly  two-thirds  of  all  risks  discovered 


DACS  is  a  Department  of  Defense-sponsored  Information  Analysis  Center  managed  by  the  Defense  Technical 

Information  Center  (DTIC). 

3  Appendix  B  provides  an  overview  of  the  ATAM.  For  a  complete  description,  see  [Clements  2002], 

®  Architecture  Tradeoff  Analysis  Method  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mel¬ 

lon  University. 

®  ATAM  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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were  risks  of  omission  (for  example,  technical  investigations  not  performed  or  architectural  deci¬ 
sions  either  not  made  or  not  captured)  [Bass  2006].  This  observation  suggests  that  architecture 
evaluators  must  be  experienced  enough  to  analyze  a  software  architecture  in  detail,  probing  be¬ 
yond  the  information  that  is  typically  presented. 

An  analysis  of  the  findings  from  the  ASSIP  data-gathering  efforts  mentioned  earlier  pointed  to 
software  architecture  as  a  problem  area,  consistent  with  the  studies  noted  above.  For  instance,  the 
survey  of  acquisition  professionals  revealed  a  general  impression  that  prime  contractors’  software 
architecture  abilities  were,  at  best,  about  average  [Kasunic  2004],  suggesting  a  need  for  rigorous 
evaluation  of  software  architectures  to  reduce  program  risk.  Yet,  according  to  the  BFIs  and  the 
PEO  interviews,  staff  skills  within  PMOs  were  not  adequate  to  evaluate  software  architectures 
[Blanchette  2005],  [Keeler  2005],  Thus,  software  architecture  is  an  acknowledged  good  practice 
in  SIS  programs,  yet  it  is  one  that  is  rarely  executed  effectively  or  evaluated  rigorously. 

Such  a  situation  is  a  case  of  acquirer/supplier  skill  mismatch.  As  depicted  in  Figure  2,  instances  of 
low  acquirer  capability  coupled  with  average  supplier  capability  often  lead  to  unpredictable  re¬ 
sults  and  even  to  program  failure.  Specifically,  in  the  case  of  software  architecture,  limited  exper¬ 
tise  on  the  part  of  the  Army  can  result  in  under-representation  of  stakeholder  needs  or  inadequate 
reviews  of  suppliers’  architectural  work. 
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Figure  2:  The  Acquirer/Supplier  Mismatch 

Since  the  software  architecture  underpins  the  entirety  of  a  software-intensive  system,  errors, 
omissions,  or  inflexibility  in  the  architecture  can  lead  to  problems  that  are  difficult  and  costly  to 
fix  as  the  system  evolves,  if  they  can  be  fixed  at  all. 
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In  addition  to  the  compelling  evidence  that  software  architecture  is  a  key  enabler  of  program  suc¬ 
cess,  the  Army  also  had  a  directly  relevant  example.  At  the  request  of  the  Army,  the  SEI  under¬ 
took  a  year-long  study  of  the  Force  XXI  Battle  Command  Brigade  and  Below  (FBCB2)  program 
in  2001.  As  the  Army’s  first  large-scale  deployment  of  digitized  capability,  FBCB2  is  crucial  to 
plans  for  a  networked  fighting  force.  What  the  study  team  found  was  that  FBCB2  software  was 
on  an  inappropriate  architectural  path  to  support  current  or  longer  term  operational  use  [Bergey 
2005], 

The  Honorable  Claude  Bolton,  ASA(AFT),  grasped  the  significance  of  the  various  software  archi¬ 
tecture  studies,  particularly  the  findings  of  the  FBCB2  study,  for  all  Army  acquisition  programs 
involving  software.  Through  his  AAG  representative  Dr.  James  Finnehan,  Bolton  pushed  the  SEI 
to  develop  a  proposal  for  an  AS  SIP  initiative  that  would  help  to  improve  the  use  of  sound  soft¬ 
ware  architecture  practices  in  Army  programs. 

The  proposal  generated  much  discussion  when  it  was  briefed  to  the  AAG,  both  because  of  its 
technical  nature  and  because  it  was  the  first  initiative  to  be  proposed  for  ASSIP.  Representatives 
discussed  their  own  experiences  with  software  architecture  issues  and  debated  the  potential  effi¬ 
cacy  of  the  proposal.  In  the  end,  the  AAG  determined  that  an  ASSIP  initiative  dealing  with  soft¬ 
ware  architecture  would  be  appropriate.  The  initiative  would  focus  on  improving  PMO  awareness 
and  staff  skills  in  software  architecture. 

The  remainder  of  this  report  discusses  the  ASSIP  software  architecture  initiative.  Section  4  re¬ 
views  the  SAI  and  its  implementation.  Section  5  reviews  the  SAI  results  to  date,  including  the 
lessons  learned.  Fastly,  Section  6  discusses  ongoing  work. 
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4  The  ASSIP  Software  Architecture  Initiative 


Beginning  in  FY04,  the  ASSIP  started  building  an  organic  software  architecture  capability  within 
the  Army  acquisition  community.  The  purpose  of  the  initiative,  as  stated  in  the  SSIMP,  was  to 
train  a  cadre  of  Army  technical  professionals  in  the  techniques  of  software  architecture  evaluation 
to  better  support  the  acquisition  of  SIS.  Additionally,  the  initiative  sought  to  introduce  architec¬ 
ture  evaluation  in  Army  policy  and  infrastructure  by  piloting  architecture  evaluation  techniques 
on  three  selected  programs4. 

ASSIP  is  working  to  ingrain  both  the  knowledge  and  practice  of  software  architecture  techniques 
into  the  culture  of  Army  acquisition.  Each  successive  SSIMP  has  defined  new  tasks  within  the 
software  architecture  initiative  that  build  on  the  accomplishments  of  the  previous  year.  In  FY05, 
the  ASSIP  sponsored  additional  courses  in  the  software  architecture  curriculum  and  three  addi¬ 
tional  architecture  evaluations5.  Further,  to  help  ASSIP  architecture  students  apply  their  new 
skills,  ASSIP  began  recruiting  Army  personnel  who  had  received  architecture  training  through  the 
SAI  and  were  SEI-authorized  ATAM  evaluators  to  participate  on  the  evaluation  teams.  The  FY06 
SSIMP  software  architecture  initiative  included  another  three  architecture  evaluations  that  in¬ 
cluded  Army  personnel  and  an  advanced  series  of  courses  from  the  SEI  software  product  line  cur¬ 
riculum.  6 

The  following  sections  describe  the  software  architecture  training  and  software  architecture 
evaluations  in  more  detail. 


4.1  SOFTWARE  ARCHITECTURE  TRAINING 

The  first  step  in  the  SAI  was  to  train  qualified  Army  technical  professionals.  The  SEI  established 
a  special  offering  of  its  publicly  available  software  architecture  curriculum  and  delivered  it  at  the 
Army  SECs.  The  curriculum  consists  of  six  courses,  as  depicted  in  Figure  3. 


4  U.S.  Army:  Fiscal  Year  2004  Strategic  Software  Improvement  Master  Plan.  September  2003. 

5  U.S.  Army:  Fiscal  Year  2005  Strategic  Software  Improvement  Master  Plan.  October  2004. 

6  U.S.  Army.  Fiscal  Year  2006  Strategic  Software  Improvement  Master  Plan.  October  2005. 
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Figure  3:  The  SEI  Software  Architecture  Curriculum 

Additional  curriculum  information,  including  course  descriptions,  may  be  found  at 
http://www.sei.cmu.edu/activities/architecture/certificatejirogram.html. 

The  ASSIP  offering  of  the  curriculum  was  conducted  with  the  full  rigor  of  the  publicly  offered 
version.  The  SEI  enforced  the  normal  prerequisites  for  each  course.  Course  content  and  material 
was  identical  to  that  in  the  SEI’s  regular  public  offerings,  and  the  same  qualified  instructors  who 
teach  the  public  sessions  taught  the  courses.  Students  in  ASSIP-sponsored  training  earned  certifi¬ 
cates  as  depicted  in  Figure  3  for  successful  completion  of  course  sequences. 

The  ASSIP  paid  for  the  courses,  which  were  delivered  at  the  various  SECs.  Students’  home  or¬ 
ganizations  paid  for  personnel  time  and  any  temporary  duty  (TDY)  expenses  (such  as  travel  costs) 
associated  with  attendance.  Holding  the  courses  at  the  SECs  minimized  TDY  expenses  for  a  por¬ 
tion  of  each  class  since  each  software  center  is  in  close  proximity  to  several  PMOs  and  PEOs. 

In  each  of  FY04  and  FY05,  30  slots  were  available  to  personnel  involved  in  Army  acquisition  or 
acquisition  support  roles  (due  to  the  nature  of  the  training,  only  15  slots  were  available  in  each 
year  for  the  more  advanced  ATAM  evaluator  and  leader  training  courses).  The  ASSIP  equitably 
allocated  slots  among  the  software  centers,  PEOs,  and  PMOs.  Since  the  SECs  are  positioned  to 
provide  broad-based  evaluation  support,  ASSIP  made  the  limited  slots  for  ATAM  Evaluator  and 
Leader  courses  available  to  them  first;  PEOs  and  PMOs  were  able  to  take  advantage  of  the  few 
slots  not  filled  by  SEC  personnel. 
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The  value  of  the  courses  has  been  recognized.  In  late  2005,  the  SEC  at  the  Communications- 
Electronics  Lifecycle  Management  Command  (C-E  LCMC)  at  Fort  Monmouth  asked  the  SEI  to 
provide  an  offering  of  the  software  architecture  curriculum  especially  for  the  Fort  Monmouth 
community. 


4.2  SOFTWARE  ARCHITECTURE  EVALUATIONS 

Beginning  in  FY04,  the  AAG  decided  to  sponsor  a  series  of  architecture  evaluations  using  the  SEI 
method  known  as  AT  AM.  The  goals  of  the  evaluations  were 

(a)  to  begin  institutionalizing  software  architecture  practices  by  showing  their  value  in  real  Army 
acquisition  programs  and  (b)  to  provide  opportunities  for  ASSIP-trained  Army  personnel  to  gain 
experience  in  applying  the  concepts  of  the  architecture  training. 

A  total  of  two  ATAM  evaluations  and  one  SEI  Quality  Attribute  WorkshopSM  (QAWSM)  method7 
were  conducted  in  FY04.  ASSIP  intended  these  initial  pilot  sessions  to  demonstrate  the  usefulness 
of  the  methods  to  the  Army  in  general  while  providing  valuable  technical  feedback  to  each  of  the 
participating  programs.  The  FY04  evaluations  were  staffed  entirely  by  skilled  evaluators  from  the 
SEI.  In  FY05  and  FY06,  the  ATAM  and  QAW  evaluations  were  led  and  staffed  by  qualified  SEI 
personnel.  In  addition,  one  to  two  Army  personnel  were  included  on  each  ATAM  evaluation 
team;  other  Army  personnel  participated  as  observers. 

Responsibility  for  nominating  candidate  programs  fell  to  the  AAG.  After  the  success  of  the  initial 
pilots  in  FY04,  the  AAG  representatives  nominated  several  programs  as  candidates  for  the  ATAM 
evaluations  and  QAWs.  However,  the  SEI  found  that  many  nominated  programs  were  not  yet 
ready  for  an  evaluation,  causing  frustration  both  for  the  programs  and  for  the  evaluation  teams. 
Consequently,  the  SEI  developed  a  method  for  selecting  among  the  candidate  programs  in  a  fair 
and  impartial  manner  that  also  included  pre-screening  of  programs  to  ensure  readiness.  In  some 
cases,  a  QAW  was  conducted  while  a  program  prepared  for  an  ATAM  evaluation. 

Similarly,  the  SEI  developed  a  way  of  selecting  ASSIP-trained  individuals  to  participate  on 
evaluation  teams.  These  selection  processes  are  discussed  below. 

The  ASSIP  paid  the  cost  of  the  ATAM  or  QAW.  Labor  and  TDY  costs  for  Army  personnel  to 
participate  on  the  teams  were  paid  by  either  the  PMO  hosting  the  ATAM  architecture  evaluation 
or  the  individuals’  home  organization  (or  both).  When  qualified  evaluators  were  not  available 
from  the  PMOs  under  the  cognizance  of  the  associated  PEO  or  supporting  SEC,  volunteers  were 
sought  from  other  external  Army  commands. 


7  The  Quality  Attribute  Workshop  is  described  in  a  2003  technical  report  [Barbacci  2003], 
SM  Quality  Attribute  Workshop  and  QAW  are  service  marks  of  Carnegie  Mellon  University. 
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4.2.1  The  ATAM  Program  Selection  Process 

Introducing  new  techniques  into  any  environment  increases  risks  for  misunderstandings  and  poor 
outcomes.  To  minimize  these  risks,  the  authors  of  this  report  developed  a  process  for  selecting 
programs  that  focused  on  clearly  communicating  the  requirements  for  receiving  an  ASSIP- 
sponsored  ATAM  to  the  PMOs  and  ensuring  their  commitment  to  proceed.  In  addition,  the  proc¬ 
ess  sought  to  distribute  ATAM  opportunities  as  fairly  as  possible  across  PEOs.  Figure  4  depicts 
the  process. 


Figure  4:  The  Program  Selection  Process 

Two  senior  SEI  technical  professionals,  with  experience  in  the  ATAM  evaluation  requirements 
and  familiarity  with  Anny  programs  in  general,  contacted  the  designated  representative  for  each 
program  nominated  by  the  AAG  representatives  to  assess  the  extent  to  which  the  programs  met 
the  minimum  selection  criteria.  Those  that  did  not  meet  the  minimum  criteria  were  excluded  from 
further  consideration.  The  SEI  staff  members  then  ranked  the  remaining  programs  according  to 
the  weighted  selection  criteria. 


Table  1  shows  the  minimum  criteria  and  rationale.  The  minimum  selection  criteria  reflect  the  ba¬ 
sic  ability  of  a  PMO  to  commit  to  executing  the  process  with  the  required  program  stakeholders 
and  the  availability  of  a  documented  software  architecture  for  evaluation. 
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Table  1:  Minimum  Program  Selection  Criteria 


Minimum  Selection  Criteria  (Unweighted) 

Rationale 

PEO  and  PMO  commitment 

PEOs  and  PMOs  must  commit  to  actively  participate  in 
the  evaluation 

Availability  of  stakeholders 

ATAM  evaluations  are  stakeholder  driven,  thus  stake¬ 
holders  must  be  willing  and  able  to  participate 

Availability  of  program  artifacts  and  architecture  docu¬ 
mentation 

An  architecture  must  be  documented  to  be  evaluated 

PMO/supplier  can  contractually  accommodate  ATAM 
engagement 

If  an  architecture  evaluation  was  not  specified  in  the 
contract  initially,  the  supplier  and  PMO  must  have  some 
understanding  to  allow  for  the  engagement  to  take  place 

Supplier  commitment  including  availability  of  chief  soft¬ 
ware  architect 

The  architect  must  be  available  to  describe  the  architec¬ 
ture  and  how  it  supports  stakeholder  needs 

The  weighted  criteria,  shown  in  Table  2,  attempt  to  discriminate  among  candidates  in  part  based 
on  previous  participation  with  the  ASSIP  or  the  SET  To  encourage  broad  application  of  architec¬ 
ture  techniques,  new  participants  were  viewed  more  favorably  in  this  regard.  Additionally,  pro¬ 
grams  were  favored  in  instances  where  the  anticipated  payoff  to  the  Army  (as  subjectively  deter¬ 
mined  by  the  SEI)  was  relatively  higher.  Lastly,  the  ability  to  schedule  an  AT  AM  evaluation  that 
met  both  program  and  SEI  constraints  also  weighed  in  a  program’s  favor.  Appendix  C  depicts  an 
example  scoring  sheet. 


Table  2:  Weighted  Program  Selection  Criteria 


Weighted  Selection  Criteria 

Rationale 

PEO  first-time  participant 

Yes  =  5  /  No  =  0 

To  encourage  broad  participation,  preference  was  given  to 
organizations  becoming  involved  in  ASSIP-related  improve¬ 
ment  activities  for  the  first  time. 

PEO  or  PMO  is  willing  to  pay  travel  costs  for  two 
Army  evaluators 

Yes  =  10  /  No  =  0 

In  the  event  that  travel  costs  were  associated  with  the  partici¬ 
pation  of  Army  evaluation  team  members,  preference  was 
given  to  PMOs/PEOs  that  were  willing  to  pay  those  expenses 
(or  at  least  share  them). 

Program  first-time  participant 

Yes  =  5  /  No  =  0 

To  encourage  broad  participation,  preference  was  given  to 
organizations  becoming  involved  in  ASSIP-related  improve¬ 
ment  activities  for  the  first  time. 

Contractor  first-time  participant 

Yes  =  5  /  No  =  0 

To  encourage  wide  exposure  of  architecture  evaluation  tech¬ 
niques,  preference  was  given  to  programs  whose  prime  con¬ 
tractor  had  no  prior  experience  with  such  techniques. 

Pay-off  potential  to  Army 

1  to  10,  where  10  is  highest  favorable  rating 

Preference  was  given  to  programs  that  were  especially  criti¬ 
cal  to  Army  modernization  efforts,  since  such  programs  had  a 
great  need  to  be  architecturally  cohesive  with  each  other. 

Opportune  timing/scheduling 

1  to  10,  where  10  is  highest  favorable  rating 

Preference  was  given  to  programs  that  were  able  to  commit 
to  scheduling  that  was  most  compatible  with  availability  of 
evaluation  teams. 

After  selecting  the  two  highest  ranked  programs,  the  SEI’s  technical  staff  confirmed  each  pro¬ 
gram’s  availability  and  commitment  and  then  forwarded  the  programs  for  the  concurrence  of  SEI 
management.  Upon  approval  of  the  programs,  the  SEI  staff  began  the  process  of  selecting  Army 
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personnel  to  participate  on  each  team.  Concurrently,  SEI  management  assigned  Lead  Evaluators, 
who  took  charge  of  subsequent  discussions  with  the  selected  programs. 


Following  successful  completion  of  the  selection  process,  the  AT  AM  evaluations  proceeded  as 
usual,  beginning  with  the  Phase  0  discussions  conducted  by  the  AT  AM  team  leader. 


4.2.2  ATAM  Team  Member  Selection  Process 


The  selection  process  for  Army  team  members  followed  a  similar  logic  as  the  program  selection 
process.  Figure  5  depicts  the  personnel  selection  process. 


Exit  Condition 


Figure  5:  The  Personnel  Selection  Process 

The  minimum  requirements  for  any  ATAM  evaluation  team  member  are  successful  completion  of 
the  Software  Architecture:  Principles  and  Practices  and  the  ATAM  Evaluator  Training  courses. 
Elowever,  since  the  ATAM  evaluation  is  fundamentally  a  team-based  method  (not  just  in  the 
sense  of  the  evaluation  team  but  also  in  the  sense  of  the  evaluators,  the  PMO,  and  the  stake¬ 
holders),  selecting  evaluation  team  members  becomes  crucially  important  for  the  effective  opera¬ 
tion  of  the  evaluation  team  and  for  the  overall  success  of  the  evaluation.  Consequently,  the  AS  SIP 
instituted  additional  gating  factors  to  help  ensure  that  Army  participants  would  be  well  matched 
both  to  the  team  and  to  the  task. 


As  shown  in  Table  3,  minimum  criteria  for  participants  emphasized  technical  competence  in  gen¬ 
eral  architecture  techniques  as  well  as  in  the  program  domain,  availability  to  participate  (based  on 
preliminary  schedules),  freedom  from  conflicts  of  interest,  personal  preparedness,  and  availability 
to  assist  in  writing  the  final  report. 
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Table  3:  Minimum  Personnel  Selection  Criteria 


Minimum  Selection  Criteria 

Rationale 

Knowledgeable  about  domain 

Individuals  who  have  some  knowledge  of  the  technical 
domain  can  be  more  insightful  about  evaluating  relevant 
architectures. 

Software  architecture  expertise 

Skill  in  developing  software  architectures  is  essential  to 
being  able  to  evaluate  them. 

Acceptable  to  key  stakeholders 

On  very  rare  occasions,  an  evaluated  program  may 
perceive  a  given  individual  to  be  unacceptably  biased. 

Willing  to  be  on  ATAM  Team  and  available  in  the  time 
frame  needed 

Individuals  must  have  both  interest  and  availability  to 
participate. 

No  known  conflict  of  interest 

Individuals  must  be  free  of  conflicts  of  interest  (including, 
but  not  limited  to,  participation  in  a  competing  program). 

Army  evaluator  has  laptop  computer  and  understands 
follow-on  commitment  to  assist  in  writing  final  report 

Individuals  must  commit  to  being  full  participants  on  the 
team  and  come  equipped  with  the  resources  to  perform. 

The  weighted  criteria  shown  in  Table  4  favored  individuals  with  a  knowledge  of  the  specific  pro¬ 
gram.  Positive  feedback  from  an  ATAM  instructor  also  weighed  in  an  individual’s  favor.  As  in 
the  program  selection  process,  new  participants  were  favored,  as  were  participants  with  a  commit¬ 
ted  funding  source.  Lastly,  individuals  who  had  completed  ATAM  Leader  training  and  who 
planned  to  become  Lead  Evaluators  were  favored  because  participation  in  an  ATAM  evaluation  is 
one  of  the  important  criteria  for  becoming  an  ATAM  Lead  Evaluator. 

Table  4:  Weighted  Personnel  Selection  Criteria 


Weighted  Selection  Criteria 

Rationale 

Knowledgeable  about  program 

1  to  10,  where  10  is  highest  favorable  rating 

Individuals  with  some  knowledge  of  the  program  can  be 
even  more  insightful  than  those  with  domain  knowledge. 

Positive  input  from  ATAM  instructor 

1  to  10,  where  10  is  highest  favorable  rating 

Candidates  who  were  deemed  especially  good  students 
were  preferred. 

SEC  or  PM  willing  to  pay  labor  costs 

Yes  =  10/ No  =  0 

In  the  event  that  travel  costs  were  associated  with  the 
participation  of  Army  evaluation  team  members,  prefer¬ 
ence  was  given  to  individuals’  home  organizations  that 
were  willing  to  pay  those  expenses  (or  at  least  share 
them). 

SEC  is  first-time  participant 

Yes  =  5  /  No  =  0 

To  encourage  broad  participation,  preference  was  given 
to  individuals  who  represented  SECs  becoming  involved 
in  ASSIP-related  improvement  activities  for  the  first  time. 

Individual  and  program  associated  with  same  PEO  or 

SEC 

Yes  =  5  /  No  =  0 

To  minimize  the  possibility  of  travel  expenses,  prefer¬ 
ence  was  given  to  individuals  who  belonged  to  the  same 
organization  that  supported  the  program. 

Completed  ATAM  Leader  training 

Yes  =  5  /  No  =  0 

Individuals  with  a  desire  to  become  Lead  Evaluators 
were  given  preference,  since  participating  on  an  evalua¬ 
tion  team  is  one  of  the  prerequisites  for  that  achieve¬ 
ment. 

The  team  member  selection  process  followed  a  flow  similar  to  that  of  the  program  selection  proc¬ 
ess.  The  same  SEI  staff  that  selected  the  programs  also  selected  the  Army  participants.  The  SEI 
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contacted  the  individuals  directly  as  well  as  their  home  organizations  as  part  of  the  screening 
process.  Selected  candidates  were  approved  by  SEI  management  and  then  assigned  to  teams. 

Following  their  selection,  Army  team  members  participated  fully  in  all  evaluation  team  activities, 
including  preparatory  work,  on-site  evaluation,  and  post-evaluation  report  development. 

Participation  on  an  AT  AM  evaluation  team  is  one  of  the  steps  individuals  must  complete  prior  to 
becoming  an  ATAM  Lead  Evaluator.  Seven  Army  personnel  have  completed  this  step  to  date. 
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5  Progress  to  Date 


Overall,  the  AS  SIP  SAI  has  succeeded  in  raising  awareness  within  the  Army  acquisition  commu¬ 
nity  of  the  value  of  software  architecture  practices,  particularly  architecture  evaluation.  This  sec¬ 
tion  and  the  next  address  results,  lessons  learned,  and  ongoing  work  associated  with  the  SAI. 

5.1  RESULTS 

As  evidenced  by  the  attendance,  the  software  architecture  training  courses  were  very  well  re¬ 
ceived  by  the  Army’s  technical  professionals.  A  total  of  64  Army  personnel  participated  in  the 
SAI  training.  Table  5  shows  the  total  participation  and  number  of  certificate  holders  by  organiza¬ 
tion  at  the  time  of  training  (some  individuals  have  since  changed  organizations).  These  organiza¬ 
tions  and  their  certificate  holders  are  now  software  architecture  assets  for  themselves  and  for  the 
broader  Army  acquisition  community. 

Table  5:  Organizations  with  ASSiP-Trained  Architecture  Professionals 


Organization 

Certificates 

Armaments  Research,  Development  and  Engineering 
Center  Software  Engineering  Center  (ARDEC  SEC) 

Total  Participants:  9 

7  ATAM  Evaluators 

7  Software  Architecture  Professionals 

Aviation  and  Missile  Research,  Development,  and  Engi¬ 
neering  Center  Software  Engineering  Directorate 
(AMRDEC  SED) 

Total  Participants:  1 1 

9  ATAM  Evaluators 

7  Software  Architecture  Professionals 

Communications-Electronics  Lifecycle  Management 
Command  Software  Engineering  Center  (C-E  LCMC 

SEC) 

Total  Participants:  12 

5  ATAM  Evaluators 

8  Software  Architecture  Professionals 

Joint  PEO  Chemical  and  Biological  Defense  (JPEO 

CBD) 

Total  Participants:  2 

1  Software  Architecture  Professional 

PEO  Ammunition  (PEO  Ammo) 

Total  Participants:  3 

2  Software  Architecture  Professionals 

PEO  Aviation  (PEO  AVN) 

Total  Participants:  3 

2  Software  Architecture  Professionals 

PEO  Command,  Control  and  Communications  Tactical 
(PEO  C3T) 

Total  Participants:  5 

1  ATAM  Evaluator 

3  Software  Architecture  Professionals 

PEO  Enterprise  Information  Systems  (PEO  EIS) 

Total  Participants:  1 

1  Software  Architecture  Professional 

PEO  Ground  Combat  Systems  (PEO  GCS) 

Total  Participants:  2 

1  ATAM  Evaluator 

1  Software  Architecture  Professional 
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Organization 

Certificates 

PEO  Intelligence,  Electronic  Warfare  and  Sensors  (PEO 
IEW&S) 

Total  Participants:  2 

2  Software  Architecture  Professionals 

PEO  Missiles  and  Space  (PEO  MS)8 

Total  Participants:  4 

2  Software  Architecture  Professionals 

PEO  Simulation,  Traininq  and  Instrumentation  (PEO 

STRI) 

Total  Participants:  2 

2  Software  Architecture  Professionals 

Tank-Automotive  Research,  Development  and  Engi¬ 
neering  Center  (TARDEC) 

Total  Participants:  8 

7  ATAM  Evaluators 

4  Software  Architecture  Professionals 

Note  that,  for  a  variety  of  reasons,  not  all  participants  were  able  to  complete  the  course  sequences 
required  to  earn  a  certificate. 


Nine  Army  programs  have  benefited  directly  from  the  application  of  architecture  evaluation 
methods.  Table  6  shows  the  Army  programs  for  which  ATAM  evaluations  and  QAWs  have  been 
conducted.  The  Common  Avionics  Architecture  System  (CAAS)  and  the  Joint  Tactical  Common 
Operational  Picture  (COP)  Workstation  (JTCW)  funded  their  evaluations  directly  rather  than 
through  the  AS  SIP,  an  indication  of  promulgation  of  software  architecture  techniques  to  the 
broader  Army  acquisition  community. 


Table  6:  Army  ATAM  Evaluations  and  QA  Ws 


Program/Project/Product 

PEO 

ASSIP 

Funded? 

FY04 

FY05 

FY06 

Manned/Unmanned 
Common  Architecture 
Program 
(MCAP) 

Aviation 

(AVN) 

Yes 

ATAM 

Aerial  Common  Sensor 
(ACS) 

Intelligence, 
Electronic  War¬ 
fare  and  Sen¬ 
sors 

(IEW&S) 

Yes 

ATAM 

Distributed  Common 

Ground  Station  -  Army 
(DCGS-A) 

Intelligence, 
Electronic  War¬ 
fare  and  Sen¬ 
sors 

(IEW&S) 

Yes 

QAW 

ATAM 

Warfighter  Information 
Network  -  Tactical 
(WIN-T) 

Command, 
Control  and 
Communica¬ 
tions  Tactical 
(C3T) 

Yes 

ATAM 

(2  Army 
Evaluators) 

Participants  were  part  of  predecessor  organizations  PEO  Tactical  Missiles  and  PEO  Air  Space  and  Missile  De¬ 
fense. 
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Program/Project/Product 

PEO 

ASSIP 

Funded? 

FY04 

FY05 

FY06 

Common  Avionics  Archi¬ 
tecture  System 
(CAAS) 

Aviation 

(AVN) 

No 

ATAM 

Integrated  Fire  Control 
(IFC) 

Missiles  and 

Space 

(MS) 

Yes 

QAW 

ATAM 

(2  Army 
Evaluators) 

One  Semi-Automated 

Forces 

(OneSAF) 

Simulation, 
Training  and 
Instrumentation 
(STRI) 

Yes 

ATAM 

(1  Army 
Evaluator  and 

1  Army  Ob¬ 
server) 

Command  Post  of  the 

Future 

(CPoF) 

Command, 
Control  and 
Communica¬ 
tions  Tactical 
(C3T) 

Yes 

ATAM 

(2  Army 
Evaluators) 

Joint  Tactical  Common 
Operational  Picture  Work¬ 
station 
(JTCW) 

Command, 
Control  and 
Communica¬ 
tions  Tactical 
(C3T) 

No 

ATAM 

Army  Battle  Command 

System 

(ABCS) 

Command, 
Control  and 
Communica¬ 
tions  Tactical 
(C3T) 

No 

QAW 

A  CASE  STUDY 

The  detailed  results  of  an  AT  AM  evaluation  are  the  property  of  the  program  office  and  further 
dissemination  of  the  information  is  subject  to  the  program  manager’s  (PM)9  discretion.  In  the  case 
of  the  WIN-T  program,  the  PM  agreed  to  allow  a  published  case  study  of  the  evaluation. 


Participants  in  the  WIN-T  ATAM  evaluation  cited  a  number  of  benefits.  Specifically,  the  AT  AM 
evaluation  helped  WIN-T  stakeholders  develop  an  appreciation  for  the  nature  and  importance  of 
the  program’s  software  effort.  In  particular,  the  evaluation  demonstrated  the  subtle  complexities 
of  the  integration  effort. 


The  ATAM  evaluation  brought  to  light  several  previously  untracked  technical  and  schedule  risks 
that  the  program  was  then  able  to  mitigate  to  reduce  their  likelihood  and  effect.  Additionally,  the 
evaluation  led  to  a  revision  of  the  software  architecture  documentation  to  improve  its  clarity. 


This  report  makes  no  distinction  among  the  roles  of  program  manager,  project  manager,  and  product  manager. 
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Participants  also  noted  that  the  format  of  the  AT  AM  evaluation  provided  a  good  opportunity  for 
communication  among  a  variety  of  program  stakeholders,  especially  among  software  developers, 
stakeholders,  and  systems  developers;  between  team  partners;  and  among  different  groups  of 
stakeholders. 

Perhaps  one  of  the  best  indications  of  the  ATAM  evaluation’s  value  is  that  the  WIN-T  program 
manager  chartered  a  software  integrated  product  team  (IPT)  to  continue  the  work  of  analyzing  the 
software  architecture  and  monitoring  the  evolving  interests  of  the  various  stakeholders. 

See  [Clements  2005a]  for  the  case  study  of  the  WIN-T  software  architecture  evaluation. 

5.2  LESSONS  LEARNED 

In  addition  to  successes,  there  are  several  lessons  to  glean  from  the  implementation  of  the  AS  SIP 
Software  Architecture  Initiative  to  date.  The  lessons  fall  into  two  categories:  (1)  those  learned 
about  barriers  to  developing  an  organic  software  architecture  capability  in  the  Army,  and  (2)  those 
learned  about  instituting  a  software  architecture  improvement  program. 

5.2.1  Barriers  to  Software  Architecture  Practices 

Those  organizations  acquiring  software  systems,  communications  systems,  or  electronics  were 
more  inclined  to  take  full  advantage  of  the  SAI  than  those  organizations  that  chiefly  acquired 
weapons  systems  (even  though  the  weapon  systems  were  likely  to  contain  significant  amounts  of 
software,  communication,  and  electronic  components).  This  was  true  principally  of  the  programs 
nominated  for  ATAM  evaluations;  personnel  interested  in  the  architecture  training  were  more 
evenly  distributed  across  the  various  acquisition  organizations. 

The  key  difference  between  organizations  appears  to  be  that  weapon  systems  acquirers  tend  to 
focus  on  big-picture  system  issues  (e.g.,  the  system  in  its  totality);  software  is  viewed  as  an  en¬ 
abler  rather  than  a  driver  of  system  behavior.  Organizations  acquiring  systems  that  have  no  func¬ 
tion  at  all  apart  from  that  provided  by  the  software  had  no  difficulty  in  appreciating  the  need  for 
software  architecture  evaluations.  Similarly,  the  persomiel  from  the  software  centers  were  twice 
as  likely  as  PEO/PMO  staffs  to  pursue  and  complete  training  as  software  architecture  profession¬ 
als.  These  outcomes  suggest  that  special  effort  may  be  required  to  reach  out  to  organizations  that 
tend  to  treat  software  as  a  less  important  implementation  detail  in  their  systems. 

5.2.2  Lessons  About  Implementing  the  Initiative 

In  the  course  of  selecting  among  the  nominated  programs,  it  became  apparent  that  some  PMs 
were  not  fully  aware  of  the  conditions  for  being  selected  as  a  participating  program  in  the  ASSIP- 
sponsored  ATAM  evaluations.  While  all  PMs  of  the  nominated  programs  were  eager  to  receive  a 
free  evaluation,  some  were  initially  unwilling  to  allow  personnel  from  their  PEO  or  SEC  to  par¬ 
ticipate,  some  objected  to  having  Army  personnel  from  unrelated  or  external  commands  on  the 
team,  and  some  were  resistant  to  participation  by  any  Anny  personnel.  The  SEI  handled  these 
objections  through  clarifying  discussions  during  the  personnel  selection  process.  However,  it  is 
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apparent  that  there  is  a  need  for  a  written  agreement  describing  the  terms  and  conditions  for  pro¬ 
gram  selection  to  ensure  all  parties  have  a  common  understanding  of  what  is  expected  of  pro¬ 
grams  that  are  selected. 

Increasingly  tight  budgets  frequently  caused  disagreements  between  PMOs  and  SECs  over  labor 
and  TDY  costs  for  Army  participants  on  the  AT  AM  evaluations,  which  greatly  complicated  the 
process  for  selecting  Army  team  members.  Sometimes,  a  cost  sharing  approach  between  the  PMO 
receiving  the  ATAM  and  the  SEC  providing  Army  evaluators  successfully  ameliorated  such  diffi¬ 
culties.  Another  effective  means  of  avoiding  the  problem  was  for  prospective  Army  evaluators  to 
include  participation  on  an  ATAM  evaluation  team  as  part  of  the  training  they  proposed  in  their 
Individual  (career)  Development  Plan  (IDP).  Such  planning  had  the  effect  of  guaranteeing  that 
funds  for  participation  would  be  available  from  the  individual’s  home  organization. 

Setting  appropriate  expectations  for  team  members  and  their  organizations  is  vital.  ATAM  evalua¬ 
tions  involve  more  than  just  on-site  activities.  Often,  pre-evaluation  teleconferences  or  meetings 
are  needed  to  prepare  and  coordinate  the  evaluation  team.  In  addition,  taking  part  in  development 
of  a  final  evaluation  report  is  an  essential  part  of  participating  on  an  evaluation  team.  Although 
these  points  are  included  during  the  ATAM  Evaluator  training,  sometimes  they  are  overlooked  if 
there  is  a  lengthy  period  between  taking  the  class  and  participating  on  a  team.  There  were  a  cou¬ 
ple  of  instances  of  misunderstanding  about  these  points  when  recruiting  participants  for  ASSIP- 
sponsored  ATAM  evaluations.  While  they  were  resolved  rather  easily,  reinforcing  the  require¬ 
ments  for  participation  up  front,  which  is  now  part  of  the  recruitment  process,  is  a  better  approach. 

Flexibility  in  scheduling  the  ASSIP-sponsored  ATAM  evaluations  was  essential.  The  dynamic 
nature  of  programs  can,  and  did,  cause  architecture  evaluations  to  be  rescheduled  due  to  unex¬ 
pected  shifts  in  program  priorities.  Changes  in  program  scope  and  schedule  also  result  in  delay  or 
even  cancellation  of  planned  evaluations.  It  is  advisable  to  have  alternate  programs  as  a  backup  if 
possible. 

The  entire  software  architecture  curriculum,  including  the  ATAM  Lead  Evaluator  course,  was 
offered  as  part  of  the  architecture  initiative.  However,  as  discussed  further  in  Section  6.1,  becom¬ 
ing  a  Lead  Evaluator  requires  satisfaction  of  several  criteria  beyond  simply  attending  the  course. 
Through  the  initiative,  individuals  were  allowed  to  take  the  ATAM  Lead  Evaluator  course  with¬ 
out  consideration  for  their  organizations’  commitment  to  follow  through  with  these  additional 
steps.  The  Lead  Evaluator  course  should  have  been  delayed  and  offered  only  to  those  individuals 
who  not  only  had  an  interest  in  becoming  Lead  Evaluators  but  who  had  the  support  of  their  or¬ 
ganizations  in  satisfying  all  the  criteria. 
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6  Ongoing  Work 


Based  on  successes  and  lessons  learned  to  date,  the  ASSIP  continues  to  focus  on  software  archi¬ 
tecture  capability  within  the  Army.  The  sections  below  describe  the  ongoing  tasks. 


6.1  ATAM  LEAD  EVALUATOR  SELECTION  PROCESS 

The  success  of  any  ATAM  evaluation  is  largely  dependent  on  the  Lead  Evaluator’s  ability  to 
make  technical  assessments,  to  lead  the  evaluation  team,  and  to  manage  the  stakeholders  of  the 
architecture  in  question.  Thus,  the  role  of  Lead  Evaluator  is  pivotal,  and  the  stringent  require¬ 
ments  of  the  selection  process  help  ensure  that  only  fully  qualified  individuals  progress  to  the 
level  of  ATAM  Lead  Evaluator. 

ATAM  Lead  Evaluators  require  not  only  specialized  technical  skills  in  the  development  and 
evaluation  of  software  architectures,  but  also  leadership  skills.  Not  all  technical  professionals  will 
posses  the  requisite  talents  to  be  successful  ATAM  Lead  Evaluators.  Consequently,  becoming  an 
SEI-authorized  ATAM  Lead  Evaluator  involves  more  than  simply  taking  the  Lead  Evaluator 
class.  Individuals  must  complete  all  of  the  software  architecture  courses,  participate  as  an  evalua¬ 
tor  on  an  ATAM  evaluation  team,  and  apply  for  and  successfully  complete  an  observation  as  a 
Lead  Evaluator.  The  observation  requires  a  special  fee. 

Becoming  a  Lead  Evaluator  is  not  a  trivial  process,  and  not  all  candidates  will  be  successful  be¬ 
cause  of  the  special  combination  of  skills  that  must  be  demonstrated.  Since  it  is  possible  to  go 
through  all  the  steps  in  the  process  without  guarantee  of  success,  becoming  a  Lead  Evaluator  re¬ 
quires  not  just  the  desire  of  an  individual  but  also  the  commitment  of  the  sponsoring  organization. 
Making  this  commitment  explicit  and  up  front  was  viewed  as  vital  for  Army  candidates  to  be  suc¬ 
cessful.  To  address  this  need,  the  SEI  developed  a  detailed  explanation  of  the  process  and  an  en¬ 
dorsement  form  specifically  for  Army  personnel  seeking  to  become  Lead  Evaluators.  The  en¬ 
dorsement  form  makes  clear  a  sponsor’s  support  of  an  individual’s  candidacy.  Appendix  D 
elaborates  the  process  and  qualifications  for  an  Army  ATAM  Lead  Evaluator.  Appendix  E  shows 
a  sample  Lead  Evaluator  endorsement  form. 

As  of  this  writing,  two  Army  personnel  who  are  SEI-authorized  ATAM  Evaluators  and  have 
completed  the  Lead  Evaluator  training  course  have  begun  the  process  to  become  SEI-authorized 
ATAM  Lead  Evaluators. 


6.2  SOFTWARE  PRODUCT  LINES 

Closely  related  to  software  architecture  practices  is  the  notion  of  software  product  lines.  As  the 
name  suggests,  software  product  line  practices  seek  to  apply  production  line  manufacturing  con¬ 
cepts  to  software  development  via  pre-planned,  strategic  software  reuse.  A  software  product  line 
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approach  can  yield  quantitative  gains  in  productivity  and  product  quality  [SEI  2006],  as  well  as 
reductions  in  development  costs. 

For  example,  the  software  product  line  implementation  for  the  U.S.  Army's  Common  Avionics 
Architecture  System  (CAAS),  which  was  not  ASSIP  funded,  resulted  in  a  number  of  benefits 
jointly  cited  by  the  Army’s  Technical  Applications  Program  Office  (TAPO)  and  by  the  CAAS 
prime  contractor.  Chief  among  those  benefits  are  projected  cost  savings  of  several  million  dollars 
[Clements  05b],  The  program  also  anticipates  substantial  improvement  in  deployment  time  over 
previous  systems  and  has  been  able  to  achieve  real,  strategic  software  reuse  as  high  as  80  percent 
[Clements  05b].  General  guidance  for  decision  makers  in  DoD  organizations  on  product  line  ac¬ 
quisition  is  described  in  [Bergey  2006]. 

Based  on  the  success  of  the  CAAS  product  line  effort,  the  AAG  elected  to  add  SEI’s  software 
product  line  course  to  the  ASSIP  curriculum.  The  curriculum  consists  of  the  following  courses: 

•  Software  Product  Lines 

•  Adopting  Software  Product  Lines 

•  Developing  Software  Product  Lines 

•  Product  Line  Technical  ProbeSM  Team  Training 

The  Software  Product  Lines  course  also  is  part  of  the  original  software  architecture  curriculum,  so 
many  Army  students  already  had  it  under  their  belts. 

The  software  product  line  curriculum  was  offered  in  a  manner  similar  to  that  of  the  software  ar¬ 
chitecture  curriculum.  As  a  follow-on  activity  in  FY07,  the  ASSIP  will  sponsor  three  product  line 
workshops  to  allow  programs  to  share  their  lessons  learned  from  implementing  product  line  archi¬ 
tectures  and  encourage  further  implementation  of  product  line  techniques  in  the  acquisition  of 
systems  of  systems. 

6.3  ASSESSING  PROGRESS 

A  key  to  any  improvement  effort  is  a  periodic  assessment  of  progress.  To  that  end,  the  FY07 
SSIMP  includes  a  task  that  will  assess  how  well  the  ASSIP  architecture  work  to  date  has  perme¬ 
ated  Army  software  engineering  practices.  The  ASSIP  will  conduct  a  software  architecture  work¬ 
shop  aimed  at  determining  the  investment  and  additional  actions  needed  to  build  and  sustain  a 
truly  organic  software  architecture  capability  within  the  Army.  This  workshop  will  be  a  hands-on 
meeting  in  which  participants  will 

•  leam  about  best  practices  and  recent  developments  in  software  architecture 

•  share  Army  experiences  in  using  software  architecture  practices,  in  particular  the  Architec¬ 
ture  Tradeoff  Analysis  Method  (AT AM)  and  Quality  Attribute  Workshop  (QAW) 


Product  Line  Technical  Probe  is  a  service  mark  of  Carnegie  Mellon  University. 
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•  discuss  ways  in  which  the  original  objective  of  the  AS  SIP  Army  Software  Architecture  Ini¬ 
tiative  can  be  achieved 

•  examine  barriers  and  enablers  to  much  broader  adoption  of  software  architecture  practices 
within  the  Army 

•  determine  the  steps  needed  to  make  software  architecture  practices  standard  practice  across 
the  Army 

The  two-day  event  will  produce  an  understanding  of  the  current  state  of  practice  within  the  Army 
regarding  software  architecture  and  will  identify  opportunities  for  building  on  the  work  done  so 
far  through  the  ASSIP  and  the  efforts  of  individual  programs. 
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7  Summary 


The  Army  recognizes  the  importance  of  software  architecture  to  its  software-intensive  systems 
acquisitions.  The  complexity  of  modem  weapon,  communication,  and  information  systems  de¬ 
mands  a  rigorous  understanding  and  application  of  software  architecture  techniques  to  reduce  risk 
in  the  acquisition  of  software-intensive  systems  and  to  increase  the  likelihood  of  successful  out¬ 
comes. 


Through  several  surveying  techniques,  the  ASSIP  discovered  that  the  Army’s  ability  to  judge  the 
software  architecture  practices  and  products  of  their  contractors  was  lacking,  while  Army  acquisi¬ 
tion  professionals  did  not  feel  their  contractor’s  software  architecture  techniques  were  particularly 
exemplary.  This  dichotomy  led  to  the  creation  of  the  ASSIP  software  architecture  initiative. 


To  date,  64  Army  technical  personnel  have  received  software  architecture  training  through  the 
ASSIP  SAI.  Several  of  them  have  augmented  their  training  with  participation  on  ATAM  evalua¬ 
tions. 


Table  7:  Architecture  Training  Synopsis  -  PEO  Staff  (including  subordinate  PMOs) 10 


Course  in 

Software 

Architecture 

Curriculum 

Number  of  Army  Personnel  Trained 

Ammo 

AVN 

C3T 

CBD 

CS& 

css 

EIS 

GCS 

IEW&S 

MS 

Soldier 

STRI 

Total 

Software 
Architecture: 
Principles 
and  Practices 

3 

2 

4 

1 

0 

1 

2 

2 

3 

0 

2 

20 

Documenting 

Software 

Architectures 

2 

2 

4 

2 

0 

1 

2 

2 

3 

0 

2 

20 

Software 
Architecture 
Design  and 
Analysis 

3 

2 

3 

1 

0 

1 

2 

2 

2 

0 

2 

18 

Software 

Product 

Lines 

2 

2 

3 

1 

0 

1 

2 

2 

2 

0 

2 

17 

ATAM 

Evaluator 

Training 

0 

0 

1 

0 

0 

0 

1 

0 

0 

0 

0 

2 

ATAM 

Leader  Train¬ 
ing 

0 

0 

1 

0 

0 

0 

1 

0 

0 

0 

0 

2 

10  At  the  time  of  training;  some  individuals  have  since  changed  organizations. 
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Table  8:  Architecture  Training  Synopsis  -SEC  Staff 


Course  in  Software 

Architecture  Curriculum 

Number  of  Army  Personnel  Trained 

AMRDEC 

SED 

ARDEC 

SEC 

C-E 

LCMC 

SEC 

TARDEC 

Total 

Software  Architecture: 

Principles  and  Practices 

8 

9 

9 

6 

32 

Documenting  Software 

Architectures 

8 

8 

9 

6 

31 

Software  Architecture  Design  and 
Analysis 

8 

8 

9 

6 

31 

Software  Product  Lines 

6 

7 

8 

4 

25 

ATAM  Evaluator  Training 

10 

7 

6 

7 

30 

ATAM  Leader  Training 

6 

6 

3 

4 

19 

In  addition,  seven  major  Army  programs  have  benefited  from  undergoing  ATAM  evaluations. 
Early  qualitative  results  indicate  that  evaluations  are  useful  in  discovering  significant  technical, 
schedule,  and  programmatic  risks  while  also  providing  a  forum  for  increased  and  improved  com¬ 
munication  between  and  among  developers  and  stakeholders  of  the  system. 

Ongoing  efforts  to  build  the  Army’s  organic  ability  to  apply  software  architecture  practices  now 
include  ASSIP-sponsored  training  in  software  product  lines,  which  is  an  advanced  concept  in  stra¬ 
tegic  software  reuse  that  requires  an  architecture-centric  approach.  Moreover,  a  few  Army  per¬ 
sonnel  want  to  pursue  SEI  authorization  as  ATAM  Lead  Evaluators,  which,  if  completed  success¬ 
fully,  will  allow  them  to  plan,  organize,  and  conduct  ATAM  evaluations  for  the  Army. 

Thus,  the  Army  is  on  its  way  toward  developing  a  culture  that  supports  software  architecture  prac¬ 
tices,  and  especially  architecture  evaluation,  as  a  valuable  tool  in  ensuring  program  success  while 
simultaneously  evolving  an  organic  capability  to  perform  such  evaluations  through  its  technical 
personnel. 
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Feedback 


Through  its  Acquisition  Support  Program  (ASP),  the  Carnegie  Mellon  Software  Engineering  In¬ 
stitute  (SEI)  is  working  to  help  improve  the  acquisition  of  software-intensive  systems  across  the 
U.S.  government.  As  part  of  its  mission,  the  SEI  is  pleased  to  discuss  the  information  presented 
here  in  more  detail.  The  SEI  is  especially  eager  to  hear  about  experiences  with  software  architec¬ 
ture  practices  in  the  other  services. 

Please  send  questions  or  comments  about  this  report  to  the  authors: 

•  Stephen  Blanchette,  Jr.  (sblanche@sei.cmu.edu) 

•  John  Bergey  (jkb@sei.cmu.edu) 

For  more  information  about  the  SETs  software  architecture  technology  or  software  product  line 
technology,  including  the  respective  curricula,  please  contact  Linda  Northrop  (lmn@sei.cmu.edu), 
director,  Product  Line  Systems  Program. 

For  additional  ASSIP  information,  please  contact  the  SEI  Chief  Software  Engineer  for  Army  Pro¬ 
grams,  Cecilia  Albert  (cca@sei.cmu.edu). 
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Appendix  A  Acronyms  and  Abbreviations 


The  alphabetical  list  below  contains  the  acronyms,  abbreviations,  and  their  meanings  as  used  in 
this  report. 

AAG 

ASSIP  Action  Group 

ABCS 

Army  Battle  Command  System 

ACS 

Aerial  Common  Sensor 

AKO 

Army  Knowledge  Online 

Ammo 

Ammunition 

AMRDEC 

Aviation  and  Missile  Research,  Development,  and  Engineering  Center 

ARDEC 

Armaments  Research,  Development  &  Engineering  Center 

ASA(ALT) 

Assistant  Secretary  of  the  Army  for  Acquisition,  Logistics,  and  Technology 

ASP 

Acquisition  Support  Program 

ASSIP 

Army  Strategic  Software  Improvement  Program 

ATAM 

Architecture  Tradeoff  Analysis  Method 

ATEC 

Army  Test  and  Evaluation  Command 
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AVN 

Aviation 

BFI 

Benchmarking  for  Improvement 

C3T 

Command,  Control  and  Communications  Tactical 

CAAS 

Common  Avionics  Architecture  System 

CBD 

Chemical  and  Biological  Defense 

C-E  LCMC 

Communications-Electronics  Lifecycle  Management  Command 

CPoF 

Command  Post  of  the  Future 

cs&css 

Combat  Support  and  Combat  Service  Support 

DACS 

Data  and  Analysis  Center  for  Software 

DAU 

Defense  Acquisition  University 

DCGS-A 

Distributed  Common  Ground  Station  -  Army 

DoD 

Department  of  Defense 

DSB 

Defense  Science  Board 

DTIC 

Defense  Technical  Information  Center 
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EIS 

Enterprise  Information  Systems 

ESC 

Electronic  Systems  Center 

FY 

Fiscal  Year 

GAO 

Government  Accountability  Office 

GCS 

Ground  Combat  Systems 

IDP 

Individual  Development  Plan 

IEPR 

Independent  Expert  Program  Review 

IEW&S 

Intelligence,  Electronic  Warfare  and  Sensors 

IFC 

Integrated  Fire  Control 

IPT 

Integrated  Product  Team 

JPEO 

Joint  Program  Executive  Office 

JTCW 

Joint  Tactical  Common  Operational  Picture  Workstation 

MCAP 

Manned/Unmanned  Common  Architecture  Program 

MILDEP 

Military  Deputy 
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MS 

Missiles  and  Space 

OneSAF 

One  Semi-Automated  Forces 

PEO 

Program  Executive  Officer 
Program  Executive  Office 

Pgm 

Program 

PM 

Program  Manager 
Project  Manager 
Product  Manager 

PMO 

Program  Management  Office 

QAW 

Quality  Attribute  Workshop 

Rep 

Representative 

SAI 

Software  Architecture  Initiative 

SEC 

Software  Engineering  Center 

SED 

Software  Engineering  Directorate 

SEI 

Software  Engineering  Institute 

SIS 

Software -intensive  systems 
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SR 

Special  report 

SSG 

Senior  Steering  Group 

SSIMP 

Strategic  Software  Improvement  Master  Plan 

STRI 

Simulation,  Training  and  Instrumentation 

TAI 

Tri-Service  Assessment  Initiative 

TAPO 

Technical  Applications  Program  Office 

TARDEC 

Tank- Auto  motive  Research,  Development  &  Engineering  Center 

TDY 

Temporary  Duty 

TN 

Technical  note 

TR 

Technical  report 

TRADOC 

Training  and  Doctrine  Command 

URL 

Universal  Resource  Locator 

U.S. 

United  States 

US  AAA 

United  States  Army  Audit  Agency 
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WIN-T 

Warfighter  Information  Network  -  Tactical 
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Appendix  B  Overview  of  ATAM  Evaluation  Method 


The  purpose  of  the  ATAM  is  to  assess  the  consequences  of  architectural  decision  alternatives  in 
light  of  quality  attribute  requirements  [Kazman  2000].  The  major  goals  of  the  ATAM  are  to 

•  elicit  and  refine  a  precise  statement  of  the  architecture’s  driving  quality  attribute  require¬ 
ments 

•  elicit  and  refine  a  precise  statement  of  the  architectural  design  decisions 

•  evaluate  the  architectural  design  decisions  to  detennine  if  they  satisfactorily  address  the 
quality  requirements 

The  ATAM  is  predicated  on  the  fact  that  an  architecture  is  suitable  (or  not  suitable)  only  in  the 
context  of  specific  quality  attributes  that  it  must  impart  to  the  system.  The  ATAM  uses  stake¬ 
holder  perspectives  to  produce  a  collection  of  scenarios  that  define  the  qualities  of  interest  for  the 
particular  system  under  consideration.  Scenarios  give  specific  instances  of  usage,  performance 
requirements,  growth  requirements,  various  types  of  failures,  various  possible  threats,  and  various 
likely  modifications.  Once  the  important  quality  attributes  are  identified  in  detail,  then  the  archi¬ 
tectural  decisions  relevant  to  each  one  can  be  illuminated  and  analyzed  with  respect  to  their  ap¬ 
propriateness. 

The  ATAM  steps  are  carried  out  in  two  main  phases.  In  the  first,  the  evaluation  team  interacts 
with  the  system’s  primary  decision  makers:  the  architect(s),  manager) s),  and  perhaps  a  marketing 
or  customer  representative.  During  the  second  phase,  a  larger  group  of  stakeholders  is  assembled, 
including  developers,  testers,  maintainers,  administrators,  and  users.  The  two-phase  approach  in¬ 
sures  that  the  analysis  is  based  on  a  broad  and  appropriate  range  of  perspectives. 1 1 

Phase  1 : 

1.  Present  the  ATAM.  The  evaluators  explain  the  method  so  that  those  who  will  be  involved 
in  the  evaluation  have  an  understanding  of  the  ATAM  process. 

2.  Present  the  business  drivers.  Appropriate  system  representative(s)  present  an  overview  of 
the  system,  its  requirements,  business  goals,  context,  and  the  architectural  quality  drivers. 

3.  Present  the  architecture.  The  system  or  software  architect  (or  another  lead  technical  per¬ 
son)  presents  the  architecture. 

4.  Catalog  the  architectural  approaches.  The  system  or  software  architect  presents  general 
architectural  approaches  to  achieve  specific  qualities.  The  evaluation  team  captures  a  list  and 
adds  to  it  any  approaches  they  saw  during  Step  3  or  learned  during  their  pre-exercise  review 


11  These  two  phases  are  sandwiched  by  two  less  intensive  phases.  Phase  0  is  a  preparation  phase  in  which  the 
evaluation  activities  are  planned  and  set  up.  Phase  3  is  a  follow-up  phase  in  which  the  final  report  is  produced  and 
opportunities  for  improving  the  process  are  considered. 
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of  the  architecture  documentation.  For  example,  “A  cyclic  executive  is  used  to  ensure  real¬ 
time  performance.”  Known  architectural  approaches  have  known  quality  attribute  properties, 
and  these  will  help  in  carrying  out  the  analysis  steps. 

5.  Generate  a  quality  attribute  utility  tree.  Participants  build  a  utility  tree,  which  is  a  priori¬ 
tized  set  of  detailed  statements  about  what  quality  attributes  are  most  important  for  the  archi¬ 
tecture  to  carry  (such  as  performance,  modifiability,  reliability,  or  security)  and  specific  sce¬ 
narios  that  express  these  attributes. 

6.  Analyze  the  architectural  approaches.  The  evaluators  and  the  architect) s)  map  the  utility 
tree  scenarios  to  the  architecture  to  see  how  it  responds  to  each  important  scenario. 

Phase  2: 

Phase  2  begins  with  an  encore  of  the  Step  1  AT  AM  presentation  and  a  recap  of  the  results  of 

Steps  2  through  6  for  the  larger  group  of  stakeholders.  Then 

1.  Brainstorm  and  prioritize  scenarios.  The  stakeholders  brainstorm  additional  scenarios  that 
express  specific  quality  concerns.  After  brainstorming,  the  group  chooses  the  most  important 
ones  through  a  voting  process. 

2.  Analyze  the  architectural  approaches.  As  in  Step  6,  the  evaluators  and  the  architect(s) 
map  the  high-priority,  brainstormed  scenarios  to  the  architecture. 

3.  Present  the  results.  A  presentation  and  final  report  are  produced  that  capture  the  results  of 
the  process  and  summarize  the  key  findings. 

Scenario  analysis  produces  the  following  results: 

•  a  collection  of  sensitivity  and  tradeoff  points.  A  sensitivity  point  is  an  architectural  decision 
that  affects  the  achievement  of  a  particular  quality.  A  tradeoff  point  is  an  architectural  deci¬ 
sion  that  affects  more  than  one  quality  attribute  (possibly  in  opposite  ways). 

•  a  collection  of  risks  and  non-risks.  A  risk  is  an  architectural  decision  that  is  problematic  in 
light  of  the  quality  attributes  that  it  affects.  A  non-risk  is  an  architectural  decision  that  is  ap¬ 
propriate  in  the  context  of  the  quality  attributes  that  it  affects. 

•  a  list  of  issues  and  a  list  of  decisions  not  yet  made.  Often  during  an  evaluation,  issues  not 
directly  related  to  the  architecture  arise.  These  may  have  to  do  with  an  organization’s  proc¬ 
esses,  personnel,  or  other  special  circumstances.  The  ATAM  process  records  these  so  that 
they  may  be  addressed  by  other  means.  The  list  of  decisions  not  yet  made  arises  from  the 
stage  of  the  life  cycle  of  the  evaluation.  An  architecture  represents  a  collection  of  decisions. 
Not  all  relevant  decisions  may  have  been  made  at  the  time  of  the  evaluation,  even  when  de¬ 
signing  the  architecture.  Some  of  these  decisions  are  known  to  the  development  team  as  hav¬ 
ing  not  been  made  and  are  on  a  list  for  further  consideration.  Others  are  news  to  the  devel¬ 
opment  team  and  stakeholders. 

Results  of  the  overall  exercise  also  include  the  summary  of  the  business  drivers,  the  architecture, 

the  utility  tree,  and  the  analysis  of  each  chosen  scenario.  All  of  these  results  are  recorded  visibly 

so  that  all  stakeholders  can  verify  they  have  been  correctly  identified. 
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The  number  of  scenarios  analyzed  during  the  evaluation  is  controlled  by  the  amount  of  time  al¬ 
lowed  for  the  evaluation,  but  the  process  insures  that  the  most  important  ones  are  addressed. 

After  the  evaluation,  the  evaluators  write  a  report  documenting  the  evaluation  and  recording  the 
information  discovered.  This  report  will  also  document  the  framework  for  ongoing  analysis  dis¬ 
covered  by  the  evaluators.  A  detailed  description  of  the  ATAM  process  can  be  found  in  [Kazman 
2000]  and  [Clements  2002], 
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Appendix  C  Example  Program  Selection  Score  Sheet 


Figure  6  depicts  an  example  of  a  scoring  sheet  for  selecting  programs  to  receive  an  ASSIP- 
sponsored  AT  AM  evaluation.  Failure  to  meet  minimum  criteria  results  in  a  net  score  of  zero  re¬ 
gardless  of  other  factors.  In  the  example  depicted,  the  inability  to  confirm  fictional  Program  7’s 
compliance  with  the  minimum  criteria  prevented  it  from  achieving  a  score  of  at  least  25  and  there¬ 
fore  being  selected  as  one  of  the  programs  to  receive  an  AT  AM  evaluation. 


ASSIP  Scoring  for  ATAM  Engagements 

PEO 

PEO-A 

PEO-B 

PEO-C 

PEO-A 

PEO-D 

PEO-D 

PEO-E 

Program 

Pgml 

Pgm2 

Pgm3 

Pgml 

Pgm5 

Pgm7 

Contractor 

Cl 

C2 

C3 

C4 

w 

C7 

V; 

! 

M  a 

PEO.  Program 
Commitment 

See  Legend 
below 

V 

V 

? 

r 

x  7 

? 

Stakeholders  Available 

See  Legend 
below 

V 

V 

-i  )  \ 

? 

Artifacts  Available 

See  Legend 

below 

V 

</ 

? 

Contractual 

Accomodation 

See  Legend 
below 

«/ 

>\, 

y 

X 

? 

Supplier  Commitment 

See  Legend 
below 

7 

AAG  &  Program  Reps 

follow  through 

See  Legend 

below 

✓ 

'  X 

✓ 

7 

w  c 

E  r 

1  1 

G  1 

H  e 

T  r 

E  i 

D  a 

1st- Time  PEO 

5  (Yes) 

0  (No) 

f 

> 

0 

0 

S 

0 

5 

PEO  PM  SEC  Pays 
Army  Evaluator 

Labor  Travel 

10 pMe) 

Nb) 

y 

10 

0 

0 

1st- Time  Program  - " 

— 1  5  (Yes)— 

0  (No)  / 

^  0 

5 

0 

0 

5 

5 

5 

1  at-  Ti  me  Gfltfukctor 

p  (Yes) 

JO  (No) 

5 

2.5 

0 

5 

5 

1  to  10 

7 

10 

5 

5 

8 

10 

Opportune  Scheduling 

1  to  10 

10 

7 

0 

SCORE 

(If  MINIMUM  Criteria  satisfied} 

22 

34.5 

0 

0 

0 

0 

0 

LEGEND 

Appears  to  comply 

V 

Compliance  questionable 

7 

Compliance  not  apparent 

X 

Figure  6:  Example  of  a  Scoring  Sheet  for  Selecting  Programs 


SOFTWARE  ENGINEERING  INSTITUTE  |  43 


44  |  CMU/SEI-2007-TR-010 


Appendix  D  ATAM  Lead  Evaluator  Criteria  for  Army 
Candidates 


Qualifications  and  Procedure  to  become  an  Army  Lead  ATAM  Evaluator 

To  qualify  as  an  SEI-authorized  ATAM  Lead  Evaluator,  an  Army  candidate  must 
complete  the  following  steps: 

1.  Obtain  an  ATAM  Evaluator  Certificate  by  completing  the  following  two  courses: 

•  Software  Architecture:  Principles  and  Practices 

•  ATAM  Evaluator  Training 

2.  Complete  the  following  courses  from  the  SEI  Software  Architecture  Curriculum: 

•  Documenting  Software  Architectures 

•  Software  Architecture  Design  and  Analysis 

3.  Participate  as  an  ATAM  Evaluator  on  at  least  one  ATAM  evaluation  team  for  an 
Army  program  and  receive  a  positive  endorsement  from  the  ATAM  team  leader. 

[Note:  Step  3  may  be  completed  before  Step  2.] 

4.  Obtain  management  endorsement  to  become  an  Army  Lead  ATAM  Evaluator  from 
your  immediate  supervisor  or  organizational  sponsor 

[This  step  of  the  process  is  unique  to  the  Arms'.  A  template  for  the  endorsement  is  available  upon  request 
from  Stephen  Blanchette  <sblanche  (a1  sei.cmu.edu>  or  John  Bergey  cikbtesei. cmu.edu>  of  the  SEI.] 

5.  Complete  a  preparatory  leadership  training  course  after  obtaining  endorsement: 

•  AT  AM  Leader  T  raining 

6.  Submit  an  Application  for  ATAM  Observation*  to  the  SEI  for  approval.  The 
purpose  of  an  ATAM  Observation  is  to  enable  an  experienced  ATAM  Lead  Evaluator 
from  the  SEI  (the  Observer)  to  observe  and  evaluate  a  candidate  ATAM  Lead  Evaluator 
(the  Candidate)  as  he  or  she  conducts  a  software  architecture  evaluation  using  the 
ATAM. 

[An  application  form  is  available  upon  request  from  Larry  Jones  <leitosei.cmu.edu>  of  the  SEI.] 

7.  Lead  a  software  architecture  evaluation  using  the  ATAM  while  being  observed. 
The  Observer  will  rate  the  Candidate's  performance  during  the  evaluation  and  submit 
an  observation  report  to  the  SEI  ATAM  Board  with  a  recommendation  whether  or  not 
the  Candidate  should  receive  the  SEI  ATAM  Lead  Certificate. 

8.  Upon  recommendation  of  the  SEI  ATAM  Board,  the  Candidate  will  be  awarded  an 
SEI  Lead  ATAM  Evaluator  Certificate. 


There  is  a  fee  associated  with  an  ATAM  Observation.  Payment  of  the  fee  does  not  guarantee  the 
candidate  will  receive  the  SEI  ATAM  Lead  Evaluator  Certificate.  The  SEI  ATAM  Board  will  decide  the 
outcome  of  the  Observation  and  notify  the  candidate. 


Version  1.1,  8  December  2005 
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Appendix  E  Army  ATAM  Lead  Evaluator  Endorsement  Form 


ENDORSEMENT  FORM 
Army  Candidates  (or  ATAM  Lead  Evaluator 

INSTRUCTIONS:  Rll  In  highlighted  fields  and  print  on  office  letterhead.  Alternatively,  retype 
on  office  letterhead.  Return  completed  and  signed  endorsement  forms  to:  Ceci  Albert, 
Software  Engineering  Institute.  NRECA  Building,  Suite  200, 4301  Wilson  Boulevard.  Arlington, 
VA  22203 


Date: 


□ 


Applicant's  Full  Name: 
Organization: 
Email  Address: 
Telephone: 

Sponsor's  Full  Name: 
Organization: 
Email  Address: 
Telephone: 

Sponsor's  Role  (check  one): 


Immediate  Supervisor 


Organizational  Sponsor 


I  hereby  endorse  the  above  named  applicant  as  a  candidate  to  be  an  Architecture  Tradeoff  Analysis 
Method  (ATAM)  Lead  Evaluator  for  the  USAtmy.  I  understand  that  SE I  certification  as  an  ATAM 
Lead  Evaluator  is  contingent  upon  the  following  requirements  with  which  I  concur  as  noted. 


Concur 

Non¬ 

concur 

Candidates  must  complete  alt  the  course  requirements  including  the  ATAM 
LeaderTrainirtg  course 

f 

n/ 

Candidate  must  participate  in  an  ATAM  evaluation  as  an  ATAM  Evaluator 
and  receive  a  positive  endorsement  from  the  ATAM  Team  Leader. 

Candidate  must  have  significant  experience  in  designing  and  developing 
software-intensive  systems  and  some  familiarity  with  modem  software 
engineering  concepts 

Candidates  must  submit  an  application  to  the  SEI  for  apptwal  to  be 
observed  as  a  Lead  Evaluator 

Candidates  must  pay  a  fee  for  the  ATAM  Observation  engagement 

Candidates  must  successfully  lead  an  architecture  evaluation  using  the 
ATAM  and  receive  a  favorable  observation  report 

I  further  understand  that  due  to  the  unique  combination  of  technical  and  facilitation  skills  required  of 
Lead  Evaluators,  not  as  candidates  will  be  successful.  The  SEI  ATAM  Board  has  sole  and  final 
responsibility  for  deteimining  successful  completion  of  all  criteria. 


(Signature) 

Job  Title:  I  I 

Version  1.1,8  December  20C6 
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